
import Head from 'next/head'

<Head>
  <script>
    {
      `(function() {
         var _hmt = _hmt || [];
(function() {
  var hm = document.createElement("script");
  hm.src = "https://hm.baidu.com/hm.js?e60fb290e204e04c5cb6f79b0ac1e697";
  var s = document.getElementsByTagName("script")[0]; 
  s.parentNode.insertBefore(hm, s);
})();
       })();`
    }
  </script>
</Head>

![LangChain](https://pica.zhimg.com/50/v2-56e8bbb52aa271012541c1fe1ceb11a2_r.gif)





TiktokenText 分割器[#](#tiktokentext-splitter "永久链接至本标题")
======================================================

- 文本如何分割：按照 `tiktoken` 标记分割

- 块大小如何测量：按照 `tiktoken` 标记计算

```python
# This is a long document we can split up.
with open('../../../state_of_the_union.txt') as f:
    state_of_the_union = f.read()

```

```python
from langchain.text_splitter import TokenTextSplitter

```

```python
text_splitter = TokenTextSplitter(chunk_size=10, chunk_overlap=0)

```

```python
texts = text_splitter.split_text(state_of_the_union)
print(texts[0])

```

```python
Madam Speaker, Madam Vice President, our

```

